Inner product-based convolution has been the founding stone of convolutional neural networks (CNNs), enabling end-to-end learning of visual representation. By generalizing inner product with a bilinear matrix, we propose the neural similarity which serves as a learnable parametric similarity measure for CNNs. Neural similarity naturally generalizes the convolution and enhances flexibility. Further, we consider the neural similarity learning (NSL) in order to learn the neural similarity adaptively from training data. Specifically, we propose two different ways of learning the neural similarity: static NSL and dynamic NSL. Interestingly, dynamic neural similarity makes the CNN become a dynamic inference network. By regularizing the bilinear matrix, NSL can be viewed as learning the shape of kernel and the similarity measure simultaneously. We further justify the effectiveness of NSL with a theoretical viewpoint. Most importantly, NSL shows promising performance in visual recognition and few-shot learning, validating the superiority of NSL over the inner product-based convolution counterparts.

The authors propose to learn a custom similarity metric for CNNs together with adaptive kernel shape. This is formulated via learning a matrix M that modulates the application of a set of kernels W to the input X via f(W, X) W' M X. Structural constraints can be imposed on M to simplify optimization and minimize the number of parameters, but in its most general form it is capable of completely modifying the behavior of W. Although at test time M can be integrated into the weights W via matrix multiplication, during learning it regularizes training via matrix factorization. In addition, a variant is proposed where M is predicted dynamically given the input to the layer via a dedicated subnetwork. A comprehensive ablation analysis is provided that demonstrates that basic version of the proposed approach performs marginally better than a standard CNN with a comparable number of parameters on CIFAR-10, but the dynamic variant outperforms it by 1%.

The proposed method is very interesting in terms of technical novelty and experiments on few-shot learning. The author response has addressed the reviewers' concerns. Note that the results in Table 1 in the rebuttal indicate that the meta-learning part of the few-shot learning approach is of a marginal importance. So, it would be appreciated if the authors toned it down in the camera-ready version.

Inner product-based convolution has been the founding stone of convolutional neural networks (CNNs), enabling end-to-end learning of visual representation. By generalizing inner product with a bilinear matrix, we propose the neural similarity which serves as a learnable parametric similarity measure for CNNs. Neural similarity naturally generalizes the convolution and enhances flexibility. Further, we consider the neural similarity learning (NSL) in order to learn the neural similarity adaptively from training data. Specifically, we propose two different ways of learning the neural similarity: static NSL and dynamic NSL. Interestingly, dynamic neural similarity makes the CNN become a dynamic inference network.

Inner product-based convolution has been the founding stone of convolutional neural networks (CNNs), enabling end-to-end learning of visual representation. By generalizing inner product with a bilinear matrix, we propose the neural similarity which serves as a learnable parametric similarity measure for CNNs. Neural similarity naturally generalizes the convolution and enhances flexibility. Further, we consider the neural similarity learning (NSL) in order to learn the neural similarity adaptively from training data. Specifically, we propose two different ways of learning the neural similarity: static NSL and dynamic NSL.